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COLLABORATIVE  PLANNING  IN  NETWORK-ENABLED  CO-LOCATED  AND 
DISTRIBUTED  ENVIRONMENTS 


EXECUTIVE  SUMMARY 

Research  Requirement: 

A  defining  feature  of  Army  transformation  will  be  the  development  of  digital 
communications  capabilities  to  support  network-enabled  distributed  battle  command.  Previously 
battle  command  would  be  conducted  with  paper  maps,  acetate  overlays,  and  voice  radio 
networks.  Future  network-enabled  battle  command  should  rely  heavily  on  new  digital 
technology  that  supports  text  messaging,  shared  digital  maps  and  graphic  overlays,  and  the 
ability  to  reach  out  to  access  information  in  shared  databases.  To  support  new  equipment 
development  a  realistic  planning  task  is  required  which  can  yield  an  objective  planning 
performance  benchmark  score.  If  this  requirement  is  not  met,  the  apparent  performance 
superiority  of  one  equipment  design  over  another  could  simply  be  a  function  of  the  planning  task 
chosen,  characteristics  of  the  planning  environment,  or  skills  that  the  test  personnel  bring  to  the 
task,  rather  than  the  inherent  capabilities  of  the  equipment.  The  Reactive  Planning  Strategies 
Simulation  (REPSS)  provides  a  potential  solution  to  the  requirement  for  a  representative 
collaborative  planning  task  that  can  be  used  to  generate  baseline  and  benchmark  performance 
scores.  The  REPSS  presents  a  group  planning  and  resource  allocation  task  which  is  set  in  the 
context  of  creating  four  weekly  plans  to  acquire  food,  transportation,  and  security  personnel  to 
send  out  supply  convoys  as  part  of  a  hypothetical  four-week  humanitarian  relief  supply  effort. 

The  REPSS  might  serve  as  a  solution  to  the  requirement  for  an  equipment  design  benchmark 
performance  measure  if  it  can  demonstrate  sensitivity  to  manipulations  in  planning  task 
conditions,  and  planning  group  skills. 

Procedure: 

Groups  of  seven  were  formed.  Each  group  had  a  commander  and  three  two-person 
teams.  Twenty-two  groups  of  Soldiers  comprised  of  officers  and  non-commissioned  officers 
were  assigned  to  either  a  control  (co-located  teams)  or  experimental  (distributed  teams)  group 
collaborative  planning  task  condition.  Each  group  received  an  orientation  briefing  followed  by 
self-paced  automated  training  on  the  REPSS  task  and  interface  tools.  Following  this  train-up, 
each  group  performed  one  40-minute  and  three  20-minute  problem  solving  sessions  to  create  and 
send  relief  supply  convoys  to  four  towns  for  four  weeks.  An  automated  measure  recorded  each 
group’s  performance  success  in  terms  of  the  proportion  of  required  relief  supplies  delivered  to 
the  towns. 

Findings: 

The  results  provide  evidence  that  the  REPSS  simulation  can  be  applied  in  controlled 
experimentation  to  develop  benchmark  estimates  of  performance  against  which  manipulations  in 
task  conditions  and  planning  group  expertise  are  compared.  A  significant  relationship  was  found 
between  group  planning  condition  and  performance.  Groups  in  the  co-located  condition 
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provided  a  significantly  greater  quantity  of  relief  supplies  than  the  groups  in  the  distributed 
condition.  Participant  experience  associated  with  rank  differences  was  found  to  be  significantly 
related  to  successful  performance  for  groups  performing  in  the  distributed  planning  condition. 
Participant  group  experience  in  terms  of  previous  group  planning  experience  was  also 
significantly  related  to  successful  performance  in  the  REPSS  exercise  for  groups  performing  in 
the  distributed  planning  condition.  Likewise,  expertise  gained  through  previous  deployment 
experience  showed  a  significant  relationship  to  performance  success  for  groups  planning  in  the 
distributed  condition.  The  REPSS  was  designed  to  present  a  challenging  and  realistic  group 
planning  task.  Results  provide  evidence  that  the  goal  was  achieved,  as  over  90%  of  participants 
responding  to  the  REPSS  survey  reported  that  the  planning  exercise  could  be  useful  in  command 
group  training. 

Utilization  and  Dissemination  of  Findings: 

The  REPSS  provided  a  standard  task  requiring  essential  collaborative  planning  skills  and 
procedures.  The  results  yielded  evidence  that  the  REPSS  simulation  can  be  applied  in  controlled 
experimentation  to  develop  a  benchmark  estimate  of  performance  against  which  manipulations 
in  task  conditions  and  planning  group  expertise  are  compared.  The  findings  provide  support  for 
the  contention  that  equipment  developers  might  use  a  simplified  planning  simulation  such  as 
REPSS  to  establish  performance  benchmark  scores  for  the  comparison  of  alternative 
collaborative  planning  equipment  designs  in  the  following  capacities: 

•  Provide  pre-test  calibration  of  group  planning  skill  (covariate  to  assess  experiment 
performance).  Currently  the  Army  does  not  have  assessment  tools  to  estimate  the 
proficiency  level  of  groups  used  as  test  troops  for  new  systems  experimentation. 

•  Serve  as  a  standardized  scenario  within  battle  lab  experiments.  New  system 
development  efforts  typically  suffer  from  the  lack  of  a  standardized  scenario  that  can 
be  applied  across  a  variety  of  experiments. 

•  Serve  as  benchmark  against  which  to  test  the  delta  (difference)  in  collective 
performance  resulting  from  a  technology  insertion. 

Command  group  planning  exercises  can  be  time  consuming,  expensive,  and  infrequently 
performed.  The  REPSS  exercise  might  be  employed  as  a  half-day,  inexpensive  training  event 
that  could  be  conducted  as  frequently  as  required  to  maintain  basic  skills  in  collaborative 
planning.  The  results  suggest  that  the  REPSS  is  a  useful  research  tool  that  objectively  measures 
group  problem-solving  abilities.  One  opportunity  for  future  REPSS  research  would  involve 
assessing  the  impacts  of  feedback  interventions  (during  and  after  the  planning  actions)  on  group 
planning  performance  success. 
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COLLABORATIVE  PLANNING  IN  NETWORK-ENABLED  CO-LOCATED  AND 
DISTRIBUTED  ENVIRONMENTS 

Introduction 


Research  Need 

A  defining  feature  of  Army  transformation  is  the  use  of  digital  communications 
capabilities  to  support  network-enabled  planning  by  geographically  distributed  forces,  and  the 
delegation  of  planning  responsibilities  to  lower  echelons.  The  success  of  distributed  planning 
will  rely  heavily  on  emerging  digital  communications  technology.  Research  tools  are  needed  to 
support  the  development  of  the  emerging  technology.  Specifically,  a  standard  collaborative 
planning  task  is  needed  that  can  provide  baseline  estimates  of  system  performance  for  various 
operating  conditions,  and  equipment  designs.  Likewise,  tools  are  needed  that  can  accurately 
assess  the  contribution  of  Soldier  skills  to  system  performance,  to  provide  early  estimates  of 
personnel  requirements.  Once  a  baseline  performance  score  is  established  for  a  planning 
condition,  or  an  equipment  design,  the  score  can  serve  as  a  benchmark  standard  against  which 
alternative  conditions  or  equipment  designs  can  be  compared.  The  benchmark  provides  a 
measurement  that  can  be  used  as  a  reference  point  in  observations  to  compare  alternative 
designs,  processes,  and  personnel  staffing  arrangements. 

The  Reactive  Planning  Strategies  Simulation  (REPSS)  was  developed  by  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  to  meet  the  requirement  for  a 
research  tool  that  objectively  measures  group  problem-solving  abilities,  and  might  meet  the 
requirement  for  a  benchmark  planning  task.  The  REPSS  presents  a  group  planning  and  resource 
allocation  task  which  is  set  in  the  context  of  a  hypothetical  humanitarian  relief  supply  effort. 

The  REPSS  planning  task  requires  essential  planning  skills  and  procedures,  and  incorporates  a 
single  “goodness  of  planning  solution”  score.  Based  on  initial  research  (Sanders,  Fultz,  &  Sharp, 
2006)  the  REPSS  appears  to  be  a  reliable  simulation  that  can  be  used  to  benchmark  planning 
group  performance. 

The  REPSS  might  be  useful  as  a  benchmarking  task  for  equipment  design  comparisons  if 
it  can  demonstrate  sensitivity  to  manipulations  in  planning  task  conditions,  and  planning  group 
skills,  both  of  which  are  likely  to  affect  planning  performance.  The  first  goal  was  to  investigate 
whether  REPSS  performance  measures  were  sensitive  to  manipulations  in  planning  task 
conditions.  This  was  explored  through  the  comparison  of  collaborative  planning  performance  of 
groups  assigned  to  distributed  versus  co-located  planning  conditions.  The  second  goal  was  to 
investigate  whether  the  REPSS  performance  measures  were  sensitive  to  manipulations  in  terms 
of  planning  group  expertise.  A  key  analysis  involved  the  comparison  of  performance  for  high 
expertise  groups  composed  entirely  of  officers  with  lower  expertise  groups  composed  of  a  mix  of 
officer  and  non-commissioned  officers  (NCOs).  For  the  convenience  of  the  reader,  a  list  of  all 
acronyms  is  provided  as  Appendix  A. 
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Background 

REPSS  overview.  The  REPSS  is  a  software-driven  simulation  that  incorporates  a 
resource  allocation  task  requiring  collaborative  group  planning  to  arrive  at  a  solution.  The 
design  of  the  REPSS  builds  on  an  approach  for  developing  collaborative  planning  tasks  and 
performance  measures  set  forth  by  Lussier,  Solick,  and  Keene  (1992).  The  REPSS  task  presents 
a  hypothetical  humanitarian  relief  effort  requiring  a  group  composed  of  a  commander  and  three 
functionally  interdependent  teams  to  send  weekly  relief  supply  convoys  to  four  separate  towns 
for  four  weeks.  The  REPSS  task  incorporates  essential  battle  command  skills  requiring  effective 
group  communications,  and  the  ability  to  adapt  to  change,  as  information  must  be  shared 
between  the  commander  and  teams  to  build  the  planning  solutions.  Army  doctrine  prescribes  the 
use  of  Mission,  Enemy,  Terrain,  Troops,  Time,  Civilians  (METT-TC)  Factors  in  delineating  the 
information  essential  to  command  and  control,  for  commanders  at  all  levels  (Field  Manual  3.0, 
Operations,  Department  of  the  Army,  2001).  Specific  tasks  and  functional  relationships  were 
incorporated  into  the  REPSS  system  to  exercise  the  METT-TC  factors,  making  the  REPSS 
exercise  applicable  across  a  broad  range  of  current  and  future  command  and  control  (C2)  support 
systems. 

In  the  REPSS  exercise,  the  senior  officer  is  assigned  as  the  group  leader,  who  then 
divides  the  remaining  members  of  the  group  into  three  teams  (supply,  transportation,  and 
security).  All  Soldiers  participate  in  a  single  three  and  one-half  hour  REPSS  exercise  which 
includes  train-up,  performance  of  the  relief  convoy  task,  and  responding  to  exit  surveys.  Four 
simulated  weekly  relief  convoys  are  planned  and  executed  during  the  three  and  one-half  hour 
exercise.  Initial  training  is  provided  by  a  self-paced  slide  presentation. 

In  order  to  effectively  solve  the  planning  task,  each  team  requires  information  originally 
provided  to  the  other  teams  and  the  leader.  The  teams  also  depend  on  one  another  for 
intermediate  products  such  as  estimates  and  assessments.  The  analysis  and  scoring  of  REPSS 
solutions  is  done  by  automated  measures  built  into  the  REPSS  system.  The  REPSS  exercise 
provides  an  overall  estimate  of  the  “goodness”  of  the  group  planning  solution  in  terms  of  the 
quantity  of  humanitarian  relief  supplies  delivered.  Additional  performance  measures  provide 
quantitative  estimates  of  the  groups’  ability  to  synchronize  the  competing  requirements  of  the 
supply,  transportation,  and  security  teams.  The  REPSS  simulation  is  delivered  on  networked 
computers  with  one  computer  devoted  to  each  position:  commander,  supply  team,  transportation 
team,  and  security  team.  A  detailed  description  of  the  REPSS  task  may  be  found  in 
Development  of  the  Reactive  Planning  Strategies  Simulation  (REPSS)  (Sanders,  Fultz,  &  Sharp, 
2006). 


Planning  challenges.  The  Army  Technology  Objective  (ATO)  titled  Leader  Adaptability 
states  that  high  operational  tempo,  volatile  mission  demands,  and  serious  resource  constraints 
present  challenges  to  leaders  in  the  contemporary  operational  environment  and  the  future 
network-enabled  environments,  which  need  to  be  integrated  into  the  way  we  train  our  leaders 
(U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  2006).  The  REPSS 
incorporates  a  number  of  features  which  support  the  Leader  Adaptability  ATO  research  goals  to 
include: 
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•  High  operational  tempo:  Planning  groups  have  only  40  minutes  to  create  and  execute 
their  plan  for  week  one,  and  20  minutes  to  adjust  and  execute  a  plan  for  each  of  the  three 
following  weeks. 

•  Adaptive  planning:  The  planning  groups  are  given  inadequate  funds  to  fully  support  the 
relief  effort,  requiring  that  they  make  tradeoffs  between  each  of  the  teams  to  maximize 
the  group  result.  A  specific  problem  event  is  embedded  in  each  of  the  four  weeks 
requiring  teams  to  adapt  their  plans  in  response. 

•  Networked  environment  tools:  REPSS  has  been  configured  as  both  a  traditional  co¬ 
located  command  group  planning  environment  and  as  a  geographically  distributed 
environment.  Features  include  voice  communications,  text  messaging,  shared 
synchronization  matrix,  interactive  map,  and  data  files  (digital  newspapers)  that 
participants  must  access  to  gather  essential  planning  information. 

Planning  group  conditions.  A  key  issue  facing  the  future  force  is  that  Army  command 
groups  must  be  capable  of  transitioning  from  the  traditional  face-to-face  (co-located  group) 
tactical  operations  center  environment  to  collaborative  planning  across  networked 
communication  systems  (distributed  group).  “Co-located  groups”  will  be  defined  as  groups 
whose  members  work  as  interdependent  teams  in  a  common  physical  space,  whose  interactions 
include  face-to-face  communications,  as  well  as  electronic  communications  (text  messaging,  and 
shared  synchronization  matrix).  In  contrast,  “distributed  groups”  are  interdependent  teams, 
separated  by  some  degree  of  physical  space,  whose  interactions  are  mediated  through  electronic 
communications  technology  (voice  radio,  text  messaging,  and  shared  synchronization  matrix). 
The  impact  of  the  transition  to  distributed  operations  is  uncertain.  Graetz,  Kimble,  Thompson, 
and  Garloch  (1997)  suggest  that  computer-mediated  communications  associated  with  distributed 
group  collaboration  can  offer  both  advantages  and  disadvantages  compared  to  traditional  co¬ 
located  collaborative  planning,  which  can  impact  the  quality  of  planning  performance: 

•  Advantages: 

o  Compared  to  spoken  statements,  text-based  messages  are  often  composed  and 
edited  more  carefully  and  received  more  quickly. 

o  Text  messages  can  be  exchanged  simultaneously,  reducing  the  negative  effects  of 
participants  having  to  wait  their  turn  to  speak. 

o  Electronic  communications  can  reduce  the  social  anxiety  experienced  by  group 
members,  allowing  them  to  contribute  more  freely  than  in  co-located  face-to-face 
communications. 

•  Disadvantages: 

o  People  may  purposely  omit  statements  normally  offered  during  face-to-face 
conversations,  such  as  brief  utterances  indicating  agreement,  attention,  or 
understanding. 

o  With  regard  to  text  messaging,  simultaneous  collaboration  may  lead  to  “attention 
blocking,”  where  members  of  the  group  fail  to  attend  to  the  messages  of  others 
while  formulating  and  typing  their  own  responses. 
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Coordination  problems  may  become  particularly  acute  in  decision-making  tasks 
requiring  the  exchange  and  integration  of  information, 
o  Any  difficulties  collecting,  structuring,  or  integrating  information  may  elevate 
cognitive  workload  and  increase  the  likelihood  of  errors. 

Previous  ARI  research  has  suggested  that  significant  differences  exist  in  the  way 
command  group  members  perform  collaborative  planning  based  on  whether  they  operate  as  a  co¬ 
located  or  distributed  group.  However,  in  the  absence  of  a  common  task  with  which  to 
benchmark  performance  a  direct  comparison  of  performance  success  between  planning 
conditions  has  not  been  possible.  Results  from  four  Future  Combat  System  -  Command  and 
Control  (FCS  C2)  experiments  conducted  by  the  Defense  Advanced  Research  Projects  Agency 
(DARPA)  revealed  that  the  co-located  command  group  engaged  in  a  fast-paced  and  flexible 
verbal  exchange  during  simulated  engagements  (Lickteig,  Sanders,  Durlach,  Lussier,  & 

Carnahan.  >03).  In  sharp  contrast,  during  experimentation  with  distributed  command  group 
members  it  was  noted  that  communication  was  less  frequent,  and  more  closely  resembled 
sequential  staff  briefings  (Holden,  et  al.,  2005).  A  similar  relationship  was  noted  in  preliminary 
experimentation  supporting  the  development  of  the  REPSS.  Given  that  there  were  only  four 
command  groups  included  in  the  preliminary  REPSS  evaluations,  a  statistical  comparison  across 
planning  conditions  could  not  be  made.  However,  the  transcription  and  analysis  of  verbal 
communications  revealed  that  Soldiers  performing  the  REPSS  planning  task  as  a  co-located 
group  exchanged  42%  more  verbal  statements  compared  to  Soldiers  performing  the  same 
planning  task  as  a  distributed  group  (Sanders,  Fultz,  &  Sharp,  2006).  An  unanswered  question  is 
whether  additional  experimental  trials  with  REPSS  would  reveal  that  the  performance  success 
measure  (percentage  of  required  supplies  delivered)  is  sensitive  to  the  manipulation  of  planning 
conditions,  co-located  vs.  distributed. 

Planning  group  expertise.  Successful  mission  command  as  described  in  Field  Manual 
(FM)  1 ,  The  Army  rests  on  the  ability  of  commanders  to  convey  the  intent  and  concept  of 
operations,  provide  resources  adequate  to  accomplish  the  mission,  and  empower  subordinates  to 
make  decisions  while  synchronizing  their  operations  (Department  of  the  Army,  2005,  pp.  3-33). 
The  REPSS  task  was  developed  to  incorporate  essential  features  of  the  Military  Decision  Making 
Process  (MDMP)  (Department  of  the  Army,  2001)  course  of  action  development  task,  where  a 
commander  works  with  a  supporting  staff  to  generate  a  course  of  action  and  estimates  of 
required  resources.  To  develop  essential  planning  expertise,  MDMP  staff  skills  training  is 
included  in  the  formal  education  of  Army  officers,  and  assignment  to  staff  work  is  a  common 
experience  during  career  progression. 

As  the  Army  moves  toward  highly  mobile  and  dispersed  operations  there  will  be  a  greater 
demand  for  planning  and  decision  making  at  lower  echelons.  Missions  formerly  planned  at 
battalion  level  may  be  accomplished  through  planning  and  execution  at  company  level.  This 
downward  shift  of  mission  responsibilities  will  place  a  corresponding  requirement  on  lower 
echelon  personnel  to  develop  the  skills  and  expertise  required  for  successful  planning.  The 
initial  research  with  REPSS  employed  a  single  expertise  group.  Army  captains,  as  participants  to 
control  for  the  potential  impact  of  expertise  differences  across  co-located  and  distributed 
planning  conditions  (Sanders,  Fultz,  &  Sharp,  2006).  While  the  use  of  captains  as  participants 
might  represent  battalion  staff  planning  expertise,  an  alternative  expertise  group  roughly 
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corresponding  to  company  level  planning  could  be  formed  by  having  a  single  captain  supported 
by  NCOs  performing  the  REPSS  planning  task.  The  present  research  operationalized  group  rank 
as  a  continuous  variable,  the  percentage  of  officers  within  each  group,  in  assessing  the 
relationship  between  rank  and  REPSS  performance  success. 

Testing  REPSS  as  a  benchmark  measurement.  For  REPSS  to  be  useful  as  a  battle 
command  benchmark  task  it  must  demonstrate  that  its  primary  performance  measure,  percentage 
of  required  supplies  delivered  to  the  towns,  is  sensitive  to  manipulations  of  important 
collaborative  planning  variables.  The  present  research  systematically  manipulated  planning  task 
conditions  (co-located  vs.  distributed),  and  planning  group  expertise  (percentage  of  the  group 
members  that  are  officers)  to  investigate  whether  reliable  group  planning  performance 
benchmark  scores  could  be  established  that  would  show  significant  differences  between  the 
manipulation  conditions. 

Hypothesis  1 :  The  REPSS  performance  success  score  will  indicate  that  groups  performing  in  the 
co-located  planning  condition  deliver  a  significantly  higher  percentage  of  the  required  food 
supplies  compared  to  groups  performing  in  the  distributed  condition. 

Hypothesis  2:  The  REPSS  performance  success  score  will  indicate  that  groups  with  a  higher 
percentage  of  officers  will  deliver  a  higher  percentage  of  the  required  food  supplies  compared  to 
groups  composed  of  fewer  officers. 

Support  for  Hypothesis  1  will  provide  evidence  that  the  REPSS  performance  success 
measure  is  sensitive  to  differences  in  features  of  the  planning  conditions.  That  would  provide 
evidence  that  the  REPSS  performance  success  measure  could  be  used  to  benchmark  the  planning 
performance  success  rate  for  a  specific  equipment  configuration.  That  would  be  used  to  compare 
alternative  equipment  design  performance  results  against  this  benchmark  standard  to  identify  the 
payoff  in  improved  performance  associated  with  a  design  feature.  In  the  present  example  the 
REPSS  performance  success  score  for  the  co-located  group  condition  can  be  used  as  a 
benchmark  against  which  to  compare  performance  success  in  the  distributed  planning  condition. 

Support  for  Hypothesis  2  will  provide  evidence  that  the  REPSS  performance  success 
measure  is  sensitive  to  differences  in  participant  collaborative  planning  expertise.  That  would 
provide  evidence  that  the  REPSS  performance  success  measure  could  be  used  to  score  the 
planning  expertise  level  of  test  groups  prior  to  employing  them  in  the  evaluation  of  alternative 
collaborative  planning  equipment  designs.  The  group  expertise  score  could  then  be  used  as  a 
covariate  in  the  analysis  of  the  performance  of  competing  equipment  designs. 

Method 


Overview  and  Design 

Groups  of  seven  Soldiers  were  assigned  to  either  the  co-located  group  or  distributed 
group  condition.  Groups  were  assigned  to  planning  conditions  so  that  there  would  be  an  equal 
number  of  groups  assigned  to  the  co-located  and  distributed  planning  conditions.  An  effort  was 
also  made  to  assign  an  equal  number  of  high  expertise  groups  composed  entirely  of  officers  and 
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lower  expertise  groups  composed  of  a  mix  of  officer  and  NCOs  to  the  two  conditions.  For  each 
condition,  the  senior  officer  was  assigned  as  the  group  leader,  who  then  divided  the  remaining 
members  of  the  group  into  three  teams.  A  “co-located  group”  condition  was  created  by  having 
all  seven  members  perform  their  planning  tasks  within  a  common  room.  A  “distributed  planning 
group”  condition  was  created  by  assigning  the  commander,  and  each  of  the  three  two-man  teams 
to  separate  rooms  (four  rooms  total)  which  prevented  any  direct  visual  or  verbal  contact  between 
the  commander  and  teams.  The  seven  members  of  each  group  acted  as  a  commander  and  three 
two-person  teams  (supply,  transportation,  and  security)  performing  the  REPSS  collaborative 
planning  task.  Each  group  received  an  orientation  briefing  followed  by  self-paced  automated 
training  on  the  REPSS  task  and  interface  tools.  Following  this  train-up,  each  group  performed 
one  40-minute  and  three  20-minute  problem  solving  sessions  to  create  and  send  relief  supply 
convoys  to  four  towns.  Automated  measures  recorded  each  group’s  performance  success  in 
terms  of  the  quantity  of  relief  supplies  delivered  to  the  towns.  At  the  conclusion  of  the  REPSS 
exercise  the  participants  completed  a  demographic  survey,  and  an  experiment  feedback  survey. 

A  detailed  description  of  the  REPSS  apparatus  and  experimental  method  is  provided  in  Sanders, 
Fultz,  and  Sharp  (2006). 

Participants 

Participants  for  this  research  consisted  of  142  U.S.  Army  officers  and  NCOs  (one 
lieutenant  colonel,  seven  majors,  76  captains,  seven  lieutenants,  six  warrant  officers,  and  44 
NCOs)  from  Fort  Knox,  Kentucky,  Fort  Benning,  Georgia,  and  Fort  Huachuca,  Arizona.  The 
sample  also  included  one  Marine  captain.  Participants  formed  22  collaborative  planning  groups. 
Twelve  Soldiers  scheduled  for  the  experiments  were  unable  to  attend  so  that  14  groups 
performed  with  all  seven  members,  five  groups  performed  with  six  members,  two  groups 
performed  with  five  members,  and  one  group  performed  with  four  members.  The  difference  in 
participant  group  size  could  potentially  impact  performance,  and  is  addressed  in  the  results 
section  of  the  report. 

Apparatus 

The  commander  and  each  two-person  team  had  networked  computers  with  which  to 
conduct  their  collaborative  planning.  The  networked  computers  provided  the  capability  to  send 
and  receive  text  messages,  and  provided  two  shared  data  matrices  where  the  commander  and 
teams  could  develop  a  weekly  spending  plan.  The  plan  was  entered  via  on-screen  resource 
request  forms.  The  reaction  of  the  environment  to  the  allocation  of  supplies  each  week  was  sent 
to  the  command  group  members  in  the  form  of  pre-written  text  messages  indicating  losses  and 
damage  proportional  to  the  shortfall  in  supplies  delivered.  Participants  in  the  co-located 
condition  could  speak  directly  to  each  other  to  exchange  information.  Participants  in  the 
distributed  condition  were  provided  with  hand-held  single  channel  voice  radio  “walkie  talkies”  to 
support  voice  communications  within  the  group.  Fifteen  groups  located  at  Fort  Knox  used  dual¬ 
display  desk  top  computers  to  perform  the  REPSS  tasks  (11  distributed,  4  co-located).  Seven 
groups  total  at  Fort  Huachuca  and  Fort  Benning  performed  the  REPSS  exercise  in  the  co-located 
planning  condition  using  single  display  networked  laptop  computers  (ARI  portable  minilab). 
When  using  the  laptop  computer  the  group  members  would  have  to  access  the  map  by  opening  a 
window  on  the  display,  rather  than  having  the  map  displayed  constantly  on  a  separate  display. 
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This  difference  in  computer  displays  (dual-display  desk  top  computers  versus  single  display 
laptop  computers)  could  potentially  impact  performance,  and  is  addressed  in  the  results  section 
of  the  report. 

The  supply,  transportation,  and  security  teams  each  had  a  resource  request  window  on 
their  REPSS  display.  The  spreadsheet  design  minimized  calculation  requirements  by  allowing 
participants  to  simply  type  in  the  quantity  of  assets  they  need.  Figure  1  presents  the 
transportation  team  resource  request  spreadsheet  tool,  where  the  team  would  enter  the  number 
and  type  of  vehicles  (either  “armored  five-ton”  trucks,  or  unarmored  “five-ton”  trucks)  requested 
for  each  town  convoy,  and  the  route.  The  spreadsheet  automatically  calculated  the 
costs  of  the  assets  for  each  town  route  and  displayed  them  in  the  “Total”  column  when  the 
participant  pressed  the  “Calculate”  button.  This  feature  facilitated  the  iterative  adjustment  of 
plans  and  fine  tuning,  based  on  group  decisions.  When  a  team  had  finished  refining  their  plan 
for  the  week,  they  selected  the  “Submit”  button  to  enter  their  plan  and  start  the  REPSS  reaction 
process. 
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Figure  1.  Transportation  team  resource  request  spreadsheet  tool. 

Collaboration  matrix.  The  REPSS  display  provided  two  blank  14  by  11  cell  matrices 
which  were  accessed  by  selecting  a  tab  on  the  screen  tool  bar.  The  shared  matrixes  allowed  any 
member  of  the  group  to  enter  information  to  support  collaborative  planning.  In  practice  the 
matrices  were  used  to  display  and  organize  information  from  each  team  to  help  define  the 
planning  problem,  share  data  and  team  estimates,  and  document  the  weekly  planning  solutions. 
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Table  1  presents  the  collaboration  matrix  developed  in  pilot  test  trial  3  (Co-located  Group), 
which  includes  the  requirements  for  food  supplies  in  the  form  of  Humanitarian  Supply  Units 
(HSU),  and  estimates  of  citizen  morale  (CITMOR). 

Table  1 


Collaboration  Matrix  with  Headings  and  Data  Entered 
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12 
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260k 
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INDIA 
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Severe 
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40 

20 

80 
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2 

110k 

260k 

5 
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68 

34 

68 

816k 

4 

220k 

520k 

6 

7 

148 

1776k 

495k 

1170k 

Note:  14  columns  were  provided,  but  only  A  through  K,  and  rows  1  through  7  are  shown  here. 


Procedure 

Training  for  participants.  All  Soldiers  participated  in  a  single  three  and  one-half  hour 
REPSS  exercise.  Training  for  participants  consisted  of  a  10-minute  overview  briefing,  followed 
by  self-paced  training  at  the  commander  and  team  workstations.  The  self-paced  training 
consisted  of  a  digital  slide  show  which  presented  general  instructions  common  to  all  participants, 
and  also  commander  or  team-specific  instructions.  The  commander  and  team-specific 
instruction  slides  included  a  step-by-step  orientation  to  REPSS  interface,  and  an  example  of  a 
data-entry  task.  A  researcher  observed  the  participant  training  and  provided  assistance  as 
needed.  Total  time  required  for  the  overview  briefing  and  self-paced  training  was  approximately 
one  hour.  After  the  train-up  the  group  began  the  REPSS  exercise.  Groups  were  told  that  they 
could  take  a  five-minute  break  between  each  of  the  four  planning  sessions  however,  they  were 
instructed  not  to  discuss  the  exercise  during  the  break.  Most  often  groups  elected  not  to  take  a 
break.  On  several  occasions  it  was  observed  that  group  members  in  the  distributed  planning 
condition  tried  to  use  the  five-minute  break  for  a  quick  After  Action  Review,  to  share 
information  and  questions  that  they  had  difficulty  conveying  when  separated.  The  research  team 
strictly  enforced  the  “No  Discussion”  constraint. 

A  typical  exercise.  In  the  REPSS  exercise  food  supplies  in  the  form  of  humanitarian 
supply  units,  referred  to  as  “HSU”  must  be  delivered  by  convoy  from  Camp  Puller  to  each  of  the 
four  towns  (Alpha,  Echo,  India,  and  Oscar)  across  four  weeks.  An  HSU  is  a  container  holding 
food  and  medical  supplies  sufficient  to  support  the  needs  of  1 00  persons  for  one  week.  Each 
convoy  is  composed  of  supplies,  convoy  guards,  and  trucks.  If  a  planning  group  fails  to  supply  a 
town  for  one  week,  they  cannot  simply  make  up  for  this  shortfall  by  providing  twice  the  needed 
supplies  the  following  week.  Instructions  to  participants  identified  recommended  levels  of 
supply,  transportation,  and  security  assets  that  should  be  allocated  to  each  convoy.  However, 
each  group  is  only  provided  approximately  75%  of  the  funding  that  would  be  required  to  provide 
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1 00%  of  the  recommended  levels  of  support.  The  commander  and  three  teams  in  each  group 
must  collaborate  in  developing  a  spending  plan  that  maximizes  the  support  that  can  be  provided 
with  the  available  funds.  Figure  2  provides  an  overview  of  the  REPSS  humanitarian  relief 
convoy  task. 
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Figure  2.  The  REPSS  convoy  requirement  for  four  separate  convoys  each  week,  for  four  weeks. 

The  commander  should  begin  the  planning  process  by  identifying  his  strategy  for 
allocating  funds  and  establishing  planning  time  limits.  The  supply  team  should  first  estimate  the 
amount  of  HSU  to  transport  to  each  town.  The  transportation  team  would  use  this  information  to 
determine  how  many  trucks  are  needed  for  the  HSU.  The  security  team  would  then  determine 
how  many  guards  must  be  hired  for  the  supply  convoy  and  for  town  security  and  provide  that 
information  to  the  transportation  team  so  that  they  can  determine  the  number  of  trucks  needed  to 
transport  the  supply  guards  and  town  guards.  Each  team  would  calculate  the  total  cost  for  their 
contribution  and  provide  it  to  the  commander.  The  commander  would  then  calculate  the  total 
cost  of  the  Week  1  relief  effort  and  compare  the  cost  to  the  total  funds  available  for  the  four- 
week  effort.  Where  the  planned  spending  level  falls  above  or  below  the  level  of  funds  the 
commander  wishes  to  expend,  the  teams  would  have  to  adjust  their  spending.  A  great  deal  of  the 
collaborative  planning  process  involves  having  the  commander  and  three  teams  develop  a 
solution  that  matches  expensive  well-protected  convoys  to  dangerous  areas  and  less  expensive 
lightly  protected  convoys  to  safer  areas. 

Measures  Development 

Overview.  The  REPSS  provides  a  standard  collaborative  planning  task  that  yields  a 
single  quantitative  “goodness  of  planning  solution”  performance  outcome  score  that  was  used  to 
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benchmark  performance  for  alternative  planning  conditions  and  Soldier  expertise.  Five  estimates 
of  group  expertise  were  developed  from  participant  demographic  survey  data  to  assess  the 
sensitivity  of  REPSS  performance  outcomes  to  the  expertise  of  participant  groups.  One  concern 
with  hypothesis  testing  is  that  a  hypothesis  test  does  not  evaluate  the  absolute  size  of  a  treatment 
effect.  Effect  size  of  comparisons  was  estimated  using  Cohen’s  d,  which  measures  the  mean 
difference  in  terms  of  the  standard  deviation.  Using  Cohen’s  d  a  mean  difference  less  than  0.2 
standard  deviation  is  considered  a  small  effect,  greater  than  0.5  is  a  medium  effect,  and  a  mean 
difference  greater  than  0.8  standard  deviation  is  considered  a  large  effect.  Additional  measures 
addressing  plan  synchronization,  and  adaptive  planning  were  developed  to  support  REPSS 
research.  However,  these  measures  are  not  reported  in  the  body  of  the  present  report  because 
they  do  not  directly  address  the  issues  of  baseline  and  benchmark  planning  performance  success 
comparisons.  The  REPSS  plan  synchronization  measure  is  presented  in  Appendix  B.  The 
REPSS  adaptive  planning  measure  is  presented  in  Appendix  C. 

Planning  Performance  Success 

Estimating  planning  performance  success.  The  percentage  of  required  supplies  delivered 
(PRSD)  score  is  automatically  calculated  by  the  REPSS  and  can  be  provided  to  participants  as 
feedback  immediately  after  each  experiment.  The  estimate  is  the  average  percentage  of  supplies 
delivered  across  the  four  towns  and  four  weeks,  the  average  of  the  sixteen  data  points.  The 
PRSD  estimate  has  a  potential  weakness.  A  group  that  provides  100%  of  required  supplies  in 
Week  1  and  Week  2,  and  nothing  in  Week  3  and  Week  4  would  receive  the  same  score  (50%)  as 
a  group  that  consistently  delivers  50%  of  required  supplies  across  all  four  weeks.  Likewise,  a 
group  that  provides  100%  of  required  supplies  to  the  two  smaller  towns,  and  zero  to  the  two 
larger  towns  each  week  could  achieve  a  score  of  50%,  while  spending  less  funds  than  a  group 
that  provided  a  consistent  50%  delivery  rate  equally  to  all  four  towns.  For  these  reasons  a 
transformation  was  applied  where  each  of  the  sixteen  PRSD  scores  were  converted  to  its  square 
root  (SQRT)  value  and  the  sixteen  values  were  summed  to  arrive  at  the  PRSD  SQRT  score.  By 
using  the  square  root  transformation  the  impact  of  extreme  values  of  supply  delivery  on  the  total 
score  is  reduced,  and  consistency  of  delivery  will  result  in  a  higher  overall  score.  The  PRSD 
SQRT  score  provides  the  single  measure  that  can  be  used  to  benchmark  planning  performance 
success. 

An  alternative  measure  labeled  “Adaptive  Planning”  was  also  considered  which  would 
provide  an  estimate  of  how  well  groups  were  able  to  adapt  their  planning  each  successive  week, 
while  controlling  for  the  success  or  failure  encountered  in  each  prior  week.  J.  M.  Cortina 
(personal  communication,  August  4, 2006)  has  suggested  that  a  comparison  of  percentage  of 
required  supplies  delivered  across  weeks  would  not  provide  a  good  estimate  of  a  groups’  ability 
to  adapt  their  plans  successfully.  Given  that  early  planning  performance  constrains  later 
performance,  Week  1  planning  performance  must  be  more  important  than  Week  2  performance. 
Week  2  is  more  important  than  Week  3,  and  Week  3  is  more  important  than  Week  4,  thus  the 
scores  are  not  independent.  A  weighted  average  of  the  PRSD  would  thus  provide  a  better 
estimate  of  group  planning  adaptability,  where  the  weights  reflect  the  differences  in  importance 
of  each  weekly  plan.  To  generate  the  weights  to  apply  to  each  weekly  PRSD  score  the  squared 
partial  correlation  between  Week  One  and  Week  Two  was  calculated,  while  controlling  for 
planning  condition.  This  was  repeated  for  Week  Two  and  Week  Three,  and  for  Week  Three  and 
Week  Four.  The  four  calculated  weekly  weights  were  summed  and  the  percentage  contributed 
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by  each  weekly  weight  was  calculated  so  that  all  weights  would  sum  to  100%.  The  resulting 
weights  applied  to  each  week’s  PRSD  score  were:  Week  1  (0.26),  Week  2  (0.25),  Week  3 
(0.25),  and  Week  4  (0.24).  With  these  weights  the  transformation  amounts  to  no  more  than 
weighting  the  weekly  PRSD  score  by  a  constant  (about  .25),  and  results  in  scores  ranging  in 
value  from  .09  to  .23  which  cannot  be  interpreted  directly  in  terms  of  amount  of  supplies 
delivered.  Given  these  limitations  it  was  decided  to  retain  the  PRSD  SQRT  score  as  the  single 
measure  of  planning  performance  success. 

Planning  Group  Expertise  Estimates 

A  key  goal  was  to  investigate  whether  REPSS  performance  measures  were  sensitive  to 
differences  in  terms  of  planning  group  expertise.  All  Soldiers  completed  a  demographic  survey 
which  addressed:  rank,  time  since  last  wargaming  a  course  of  action  in  a  command  group, 
experience  in  performing  planning  with  the  other  members  of  the  experimental  group,  previous 
deployments,  and  experience  with  commercial  computer  applications.  Five  estimates  of  group 
expertise  were  developed  from  the  survey  data  for  comparison  against  the  PRSD  SQRT 
performance  success  score. 

Group  rank  It  was  anticipated  that  officers  might  have  more  expertise  than  NCOs  at 
collaborative  planning  tasks  given  their  training  and  experience.  The  Group  Rank  measure  was 
calculated  as  the  percentage  of  officers  within  each  participant  group.  Two  types  of  participant 
groups  were  requested,  Officer  Groups  (all  seven  positions  filled  by  officers),  and  NCO  Groups 
(one  captain  and  six  NCOs  E7  or  above).  Due  to  troop  support  constraints  it  was  not  always 
possible  to  have  the  exact  rank  group  composition  requested.  The  22  REPSS  exercises  were 
conducted  with  1 5  groups  composed  of  four  or  more  officers,  and  seven  groups  composed  of 
four  or  more  NCOs  with  a  captain  serving  as  group  commander. 

Group  planning  experience.  Recent  group  planning  experience  is  likely  to  contribute  to 
planning  expertise  and  success  on  the  REPSS  task.  The  demographic  survey  included  the 
question  “How  long  has  it  been  since  you  last  participated  in  a  group  planning  session?” 
Participant  responses  were  scored  to  indicate  those  who  had  participated  in  group  planning  in  the 
last  year,  and  those  who  had  not.  The  Group  Planning  Experience  score  is  the  percentage  of 
participants  in  a  group  who  have  participated  in  group  planning  in  the  last  year. 

Group  cohesion.  Groups  composed  of  Soldiers  who  have  experience  planning  together 
could  be  expected  to  do  better  on  the  REPSS  planning  task  compared  to  groups  where  the 
members  don’t  know  each  other.  A  question  in  the  demographic  survey  asked  “How  many 
members  of  your  experiment  group  have  you  worked  with  before  on  planning  tasks?”  The 
Group  Cohesion  score  is  the  average  of  the  group  member  responses  on  this  survey  item,  the 
average  number  of  people  in  the  group  that  the  members  have  planned  with  before. 

Group  deployment  experience.  Soldiers  who  have  been  deployed  may  have  gained 
experience  in  areas  such  as  communicating  and  rapid  decision  making  that  would  help  them  in 
accomplishing  the  REPPS  planning  tasks.  A  demographic  survey  question  asked  participants  to 
identify  whether  they  had  been  deployed.  The  Deployment  Experience  score  is  the  percentage  of 
participants  in  a  group  who  had  been  previously  deployed.  The  question  gathering  this  data  was 
added  after  the  initial  four  trials.  Data  are  available  for  Soldiers  from  the  1 8  subsequent  trials. 
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Group  computer  experience.  Expertise  in  using  computers  could  contribute  to  success  in 
performing  the  REPSS  planning  tasks.  It  should  be  noted  that  the  REPSS  was  designed  to  be 
very  simple  to  use,  so  that  computer  skills  would  not  be  required  for  successful  performance.  A 
question  in  the  demographic  survey  asked  participants  to  indicate  the  years  of  experience  they 
have  using  commercial  computer  applications.  The  Group  Computer  Experience  score  is  the 
average  number  of  years  of  computer  experience  for  the  members  of  the  group. 

Results 

The  results  provide  evidence  that  the  REPSS  simulation  can  be  applied  in  controlled 
experimentation  to  develop  a  baseline  estimate  of  performance  which  becomes  the  benchmark 
against  which  manipulations  in  task  conditions  and  planning  group  expertise  are  compared. 

They  provide  support  for  the  contention  that  equipment  developers  might  use  a  simplified 
planning  simulation  such  as  REPSS  to  establish  performance  benchmark  scores  for  the 
comparison  of  alternative  collaborative  planning  equipment  designs. 

REPSS  Sensitivity  to  Planning  Condition  Manipulations 

Planning  success  by  planning  condition.  The  key  assessment  involves  the  comparison  of 
the  co-located  condition  REPSS  performance  benchmark  score  against  the  performance  score 
attained  for  the  distributed  planning  condition.  An  independent-samples  t  test  comparing  the 
PRSD  SQRT  score  for  the  co-located  and  distributed  conditions  revealed  a  significant  difference 
between  the  two  groups  (f(20)  =  2.378, p  <  .05  [1  tailed  test]).  The  Cohen’s  statistic  d  =  1.02, 
provided  additional  evidence  of  a  large  effect  size,  providing  evidence  of  a  difference  between 
the  planning  conditions.  The  performance  mean  for  the  co-located  planning  condition 
(benchmark  score)  was  greater  ( m  =  11.5,  sd=  1.76)  than  the  mean  for  the  distributed  condition 
(m  =  9.5,  sd  =  2.13).  This  finding  provided  some  evidence  that  the  PRSD  SQRT  score  was 
sensitive  to  the  manipulation  of  planning  conditions.  The  PRSD  SQRT  score  for  the  co-located 
condition  might  prove  useful  as  a  performance  benchmark  against  which  a  performance 
decrement  for  the  distributed  planning  condition  can  be  detected. 

As  mentioned  previously,  12  Soldiers  scheduled  for  the  experiments  were  unable  to 
attend  so  that  14  groups  performed  with  all  seven  members,  five  groups  performed  with  six 
members,  two  groups  performed  with  five  members,  and  one  group  performed  with  four 
members.  An  independent  samples  t  test  was  conducted,  and  indicated  that  the  average  number 
of  Soldiers  in  the  co-located  planning  condition  ( m  =  6.55,  sd=  .82)  and  the  distributed  condition 
(m  =  6.27,  sd=  1.00)  did  not  differ  significantly  (/(20)  =  .459,  p  >  .10  [2  tailed  test]).  The 
smallest  participant  group  consisted  of  only  four  Soldiers.  An  independent  samples  t  test  was 
conducted,  indicating  that  the  PRSD  SQRT  performance  score  for  the  4-Soldier  group  (10.39, 
sd  =  0)  and  all  other  groups  (10.50,  sd  =  2.21)  did  not  differ  significantly  (/(20)  =  .963,  p  >  .10  [2 
tailed  test]).  Two-person  teams  will  continue  to  be  used  in  future  experiments,  as  a  great  deal  of 
within-team  discussions  occurred  which  should  impact  the  quality  of  solutions. 

Another  potential  confounding  variable  that  needed  to  be  explored  involves  the  different 
computer  displays  used  (dual-display  desk  top  computers  versus  single  display  laptop 
computers)  which  could  potentially  impact  performance.  An  independent-samples  t  test 
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comparing  the  PRSD  SQRT  performance  success  scores  for  co-located  groups  using  desktop 
versus  laptop  computers  did  not  find  a  significant  difference  between  the  means  of  the  two 
groups  (/(20)  =  -1 .54,  p  >  .05).  The  mean  for  the  groups  using  laptop  computers  was  not 
significantly  lower  (m  =  10.02,  sd  =2.13)  than  the  mean  of  the  groups  using  desktop  computers 
(nt  =  1 1 .50,  sd  =  2.00).  This  finding  fails  to  provide  evidence  that  performance  differences 
associated  with  the  use  of  the  single  laptop  display  contributed  to  observed  differences  in  HSU 
delivery  performance. 

REPSS  Sensitivity  to  Differences  in  Planning  Group  Expertise 

Estimates  of  planning  group  expertise  obtained  with  the  REPSS  demographic  survey 
were  compared  to  the  PRSD  SQRT  score.  Significant  results  provided  evidence  that  the  REPSS 
performance  measure  was  sensitive  to  differences  in  planning  group  expertise,  and  that 
benchmarks  might  be  established  for  these  expertise  factors  corresponding  to  desired  planning 
performance  levels.  Prior  to  analyzing  the  data  separately,  it  is  important  to  know  that  co¬ 
located  and  distributed  groups  did  not  differ  significantly  in  expertise  from  the  outset  of  this 
experiment.  The  results  of  independent-samples  t  tests  reveal  that  for  the  five  skill  variables,  the 
groups  performing  in  the  co-located  condition  differ  significantly  compared  to  groups 
performing  in  the  distributed  condition  on  only  the  Previously  of  Group  Previously  Deployed 
demographic  score.  The  percentage  of  Soldiers  who  had  previously  been  deployed  was 
significantly  greater  (r(  1 6)  =  \.79,p  <  .10  [2  tailed  test])  for  the  co-located  planning  condition 
( m  =  .92,  sd=  .11)  compared  to  the  distributed  condition  (m  =  .79,  sd=  .19).  The  rate  of 
previous  deployment  is  high  for  groups  in  both  planning  conditions,  but  the  difference  is 
significant.  As  reported  in  Table  2,  the  Percentage  of  Group  Previously  Deployed  measure  is 
significantly  related  to  performance  success  only  for  the  Distributed  group. 

An  exploratory  series  of  correlational  analyses  were  performed  to  examine  whether  group 
expertise  factors  contributed  to  performance  success  (see  Table  2).  Separate  analyses  are 
presented  for  co-located  and  distributed  groups  as  different  skill  demands  might  be  present  in  the 
two  conditions.  Given  that  the  sample  sizes  were  small  (n  =  1 1)  in  both  conditions,  results 
should  be  interpreted  with  caution  and  are  primarily  reported  here  for  exploratory  purposes  and 
to  stimulate  future  research  ideas.  The  present  research  intentionally  included  both  officers  and 
NCOs  to  provide  a  range  of  rank,  planning  experience,  and  computer  experience.  As  might  be 
expected,  some  of  these  expertise  factors  were  found  to  be  correlated.  Evaluating  the  data  from 
all  22  groups,  the  Group  Rank  measure  was  significantly  related  to  Percent  Group  Planned  in 
Last  12  Months  (r  =  .67,  p  <  .01),  and  Years  of  Computer  Experience  (r  =  .70  ,p<  .01). 

Group  rank.  Officers  might  bring  formally  trained  planning  skills  and  experience  to  the 
REPSS  task  that  contribute  to  performance  success,  while  NCOs  could  possess  these  skills  to  a 
lesser  extent.  As  shown  in  Table  2,  a  Pearson  correlation  was  calculated  to  estimate  the 
relationship  between  the  percentage  of  officers  in  each  group  and  the  PRSD  SQRT  score  for  both 
the  co-located  and  distributed  group  conditions.  The  percentage  of  officers  and  PRSD  SQRT 
score  were  not  significantly  correlated  for  the  co-located  condition.  A  moderate  significant 
correlation  was  found  for  the  distributed  condition  (r( 9)  =  .45,  p  <  .10),  which  provides  some 
evidence  that  rank  related  proficiency  contributed  to  observed  differences  in  HSU  delivery 
performance  in  the  distributed  condition.  One  concern  was  that  the  participant  rank  might  not 
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Table  2 


Expertise  Indicator  Mean,  and  Correlation  with  PRSD  SQRT  Performance  Score 


Expertise  Indicator 

Co-located 

M  t 

Distributed 

M  L 

Group  Rank  (percent  officers  in  group) 

.81  (-.20) 

.56  (.45)* 

Percent  Group  Planned  in  Last  12  Months 

.65  (-.27) 

.55  (.58)** 

Cohesion  (average  number  of  group  members  known) 

.71  (.09) 

1.11  (-.01) 

Percent  of  Group  Previously  Deployed 

.92  (-.20) 

.79  (.67)** 

Years  of  Computer  Experience 

4.03  (-.40) 

3.83  (.36) 

Notes.  */?<.10,  **  p<. 05  (all  significance  tests  are  1-tailed) 


have  been  equally  allocated  across  conditions,  and  that  this  difference  could  contribute  to  the 
higher  PRSD  SQRT  score  for  the  co-located  planning  condition.  A  Pearson  correlation  was 
calculated  to  estimate  the  relationship  between  planning  condition  and  PRSD  SQRT  scores, 
controlling  for  Group  Rank.  A  moderate  negative  correlation  was  found  (r(  1 9)  =  -.41  ,p<  .05), 
which  provides  evidence  that  the  co-located  condition  was  superior  to  the  distributed  condition 
in  terms  of  the  PRSD  SQRT  planning  success  score  even  when  controlling  for  the  potential 
contribution  of  group  member  rank. 

Group  planning  experience.  It  was  anticipated  that  group  planning  experience  could 
have  a  positive  impact  on  the  collaborative  planning  task  success.  An  estimate  of  group  planning 
experience  was  calculated  as  the  percentage  of  group  members  who  had  conducted  group 
planning  in  the  past  year.  Values  on  this  group  expertise  estimate  ranged  from  .00  (no  member 
of  the  group  had  engaged  in  group  planning  in  the  last  12  months)  to  1.00  (all  members  had 
engaged  in  group  planning,  ( m  =  .42,  sd  =  .34).  As  shown  in  Table  2,  a  Pearson  correlation  was 
calculated  to  estimate  the  relationship  between  the  Group  Planning  Experience  score  and  the 
PRSD  SQRT  score  for  both  the  co-located  and  distributed  group  conditions.  No  significant 
correlation  was  found  for  the  co-located  condition.  In  contrast,  a  significant  positive  correlation 
was  found  for  the  distributed  condition  (r(9)  =  .58,/?  <  .05).  The  findings  provided  some 
evidence  that  for  groups  performing  as  distributed  teams,  recent  experience  with  similar  planning 
tasks  contributed  to  PRSD  SQRT  planning  success. 

Group  cohesion.  If  team  members  have  planned  together  previously  they  might  be  better 
able  to  perform  the  REPSS  task  requiring  collaboration  skills.  The  Group  Cohesion  score  is  the 
average  number  of  people  in  the  group  who  have  planned  together  before.  The  values  for  the 
Group  Cohesion  score  were  m  =  1 .02,  sd=  1 .09.  A  Pearson  correlation  was  calculated  to 
estimate  the  relationship  between  the  Group  Cohesion  score  and  the  PRSD  SQRT  score  for  both 
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the  co-located  and  distributed  group  conditions.  There  was  no  significant  correlation  between 
Group  Cohesion  and  the  PRSD  SQRT  for  both  the  co-located  and  distributed  conditions.  These 
findings  failed  to  provide  evidence  that  proficiency  gained  through  having  worked  with  group 
members  before  on  similar  tasks  contributed  to  observed  differences  in  PRSD  SQRT  scores. 

Group  deployment  experience.  The  Group  Deployment  Experience  score  documents  the 
percentage  of  participants  in  each  planning  group  who  have  previously  been  deployed.  Values 
on  this  measure  ranged  from  .57  (four  members  out  of  seven  had  been  deployed)  to  1 .00  (all 
members  had  been  deployed)  ( m  =  .78,  sd=  .18).  As  shown  in  Table  2,  a  Pearson  correlation 
was  calculated  to  estimate  the  relationship  between  the  deployment  experience  and  the  PRSD 
SQRT  score  for  both  the  co-located  and  distributed  group  conditions.  No  significant  correlation 
was  found  for  the  co-located  condition.  In  contrast,  a  significant  correlation  was  found  for  the 
distributed  condition  (r(9)  =  .67,  p  <  .05).  These  findings  provided  some  preliminary  evidence 
that  groups  with  more  members  who  have  been  deployed  tend  to  be  more  successful  when 
required  to  perform  as  distributed  teams. 

Group  computer  experience.  It  was  desired  that  the  REPSS  planning  task  would  be 
sensitive  to  essential  collaborative  planning  skills  and  not  influenced  by  extraneous  factors  such 
as  general  computer  skills  proficiency.  One  question  on  the  Demographic  Survey  asked 
participants  to  indicate  their  years  of  experience  with  commercial  computer  applications.  The 
Group  Computer  Experience  score  is  the  average  number  of  years  of  computer  experience  for  the 
members  of  the  group.  The  mean  value  for  the  Group  Computer  Experience  score  was  m  =  3.92, 
sd  =  1 .37  years.  No  significant  correlations  were  found  between  the  PRSD  SQRT  score  and 
Group  Computer  Experience  for  either  the  co-located  condition  or  distributed  condition.  These 
findings  failed  to  provide  evidence  that  computer  skills  proficiency  contributed  to  observed 
differences  in  PRSD  SQRT  scores.  This  is  consistent  with  the  REPSS  design  goal  of  presenting 
a  highly  simplified  task,  an  easy  to  use  spreadsheet  interface,  and  very  simple  map  tools,  so  that 
computer  skills  would  not  be  a  contributing  factor  to  planning  performance  success. 

Potential  factors  contributing  to  planning  performance  success.  Observations  of  the 
behavior  exhibited  by  high  performing  groups  suggest  that  differences  in  planning  success  might 
be  attributable  in  part  to  several  leader  behaviors  and  the  effective  leveraging  of  planning  tool 
capabilities.  Given  that  commander  expertise  could  be  a  critical  factor  in  group  planning 
success,  it  is  important  to  identify  whether  commander  expertise  was  equivalent  between  co¬ 
located  and  distributed  groups.  The  results  of  independent-samples  t  tests  reveal  no  significant 
differences  between  co-located  and  distributed  group  commander  scores  on  the  five  skill 
variables  (rank,  planning  experience  in  last  12  months,  previous  deployment,  years  of 
commercial  computer  experience).  These  results  fail  to  provide  evidence  that  differences  in 
Commander  skills  contribute  to  the  differences  in  performance  seen  between  the  two  planning 
conditions. 

It  appears  that  collaborative  planning  was  more  successful  when  the  commander 
provided  an  early  concept  for  allocating  funds,  set  (and  enforced)  specific  planning  time  limits, 
and  required  that  teams  use  the  synchronization  matrix  to  share  information.  By  provided  an 
early  rough  spending  plan,  commanders  allowed  teams  to  calculate  early  “ball-park”  solutions  to 
their  resource  needs  which  left  more  time  for  fine-tuning  the  final  plan.  Time  management 
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appeared  to  be  a  challenge.  Many  groups  ran  out  of  time  during  the  planning  process  and  had  to 
implement  poorly  synchronized  solutions  in  order  to  meet  the  deadline.  In  general,  successful 
groups  quickly  populated  the  synchronization  matrix  with  the  key  data  and  information 
requirements  of  each  team.  The  use  of  the  synchronization  matrix  allowed  for  the  simultaneous 
rapid  and  accurate  sharing  of  information  and  allowed  the  commander  to  monitor  the  progress  of 
the  separate  teams.  For  both  distributed  and  co-located  groups  it  appeared  that  failure  to  use  the 
synchronization  matrix  led  to  slower  planning  progress,  with  sequential  question  and  answer 
exchanges  that  interrupted  team  planning,  and  required  teams  to  repeatedly  check  to  make  sure 
that  they  had  the  most  current  version  of  the  planning  numbers.  Instances  were  noted  where 
participants  made  severe  errors  when  verbally  communicating  numerical  estimates,  transposing 
numbers  and  making  order-of-magnitude  mistakes  (e.g.,  reading  “5000”  but  saying  “500”). 

Feedback  from  participants.  The  REPSS  was  designed  to  present  an  efficient  half-day 
exercise  hosted  on  commercial  computers  with  automated  training  and  performance  feedback 
that  could  support  both  new  equipment  experimentation,  as  well  as  basic  collaborative  planning 
skills  training.  Results  from  the  Demographic  Survey  provided  evidence  that  the  REPSS 
exercise  tapped  into  planning  skills  that  should  be  addressed  in  Army  training.  When  asked 
about  the  training  potential  of  REPSS,  120  of  133  participants  (over  90%)  who  responded  to  the 
survey  question  reported  that  the  planning  exercise  could  be  useful  in  command  group  training. 

Discussion 

The  REPSS  was  developed  by  ARI  to  meet  the  requirement  for  a  research  tool  that 
objectively  measures  group  problem-solving  abilities.  The  REPSS  provides  a  standard  task 
requiring  essential  collaborative  planning  skills  and  procedures.  The  results  provide  evidence 
that  the  REPSS  simulation  can  be  applied  in  controlled  experimentation  to  develop  a  baseline 
estimate  of  performance  which  becomes  the  benchmark  against  which  manipulations  in  task 
conditions  and  planning  group  expertise  are  compared.  This  provides  support  for  the  contention 
that  equipment  developers  might  use  a  simplified  planning  simulation  such  as  REPSS  to 
establish  performance  benchmark  scores  for  the  comparison  of  alternative  collaborative  planning 
equipment  designs. 

The  key  assessment  involved  the  comparison  of  the  co-located  condition  REPSS 
performance  benchmark  score  against  the  performance  score  attained  in  the  distributed  planning 
condition.  A  statistically  significant  difference  was  found  for  the  comparison,  suggesting  that 
groups  performing  in  the  co-located  condition  were  better  able  to  perform  the  collaborative 
planning  tasks,  delivering  a  greater  percentage  of  the  required  supplies.  Another  way  of  viewing 
the  results  is  to  say  that  performance  by  groups  in  the  distributed  planning  condition  falls  below 
the  benchmark  performance  score  achieved  by  groups  performing  in  the  co-located  condition. 
The  finding  provides  evidence  that  the  REPSS  PRSD  SQRT  planning  performance  measure  is 
sensitive  to  the  manipulation  of  planning  conditions. 

The  results  also  provide  some  evidence  that  three  planning  group  expertise  factors  were 
related  to  successful  performance  on  the  REPSS  exercise.  The  rank  of  the  group  members 
(percentage  of  officers  in  the  group)  showed  a  significant  and  moderate  relationship  with 
performance  success  for  groups  planning  in  the  distributed  condition.  This  suggests  that  the 
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planning  skills  associated  with  officer  expertise  do  contribute  to  observed  planning  performance 
success  when  teams  must  collaborate  through  computer-mediated  communications.  Group 
planning  experience  showed  a  significant  and  moderately  strong  relationship  to  performance 
success  for  groups  planning  in  the  distributed  condition.  Expertise  gained  through  previous 
deployment  experience  also  showed  a  significant  and  moderately  strong  relationship  to 
performance  success  for  groups  planning  in  the  distributed  condition.  This  suggests  that 
communication  or  planning  skills  developed  in  deployments  contributed  to  success  in  the 
collaborative  REPSS  planning  tasks.  These  findings  provide  evidence  that  the  REPSS  could 
have  utility  as  a  research  tool  for  investigating  the  contributions  of  specific  types  of  expertise  to 
collaborative  group  planning  performance.  Again  the  moderately  strong  relationship  found 
between  expertise  factors  and  the  PRSD  SQRT  performance  success  measure  suggests  that  with 
additional  trials  statistically  significant  relationships  might  be  demonstrated. 

Taken  together,  these  results  suggest  that  the  REPSS  might  be  a  useful  tool  for 
development  of  baseline  estimates  of  system  performance,  and  for  identifying  the  contribution  of 
Soldier  skills  to  system  performance.  With  regard  to  Hypothesis  1 ,  the  findings  do  provide  some 
evidence  matching  the  prediction  that  groups  performing  in  the  co-located  planning  condition 
would  deliver  more  food  supplies  compared  to  groups  performing  in  the  distributed  condition. 
The  finding  supports  the  use  of  REPSS  to  benchmark  the  planning  performance  success  rate  for 
a  specific  equipment  configurations,  to  identify  the  payoff  in  improved  performance  associated 
with  design  changes.  Likewise,  the  REPSS  might  be  used  as  a  standard  task  for  scoring  the 
planning  expertise  level  of  test  groups  in  order  to  factor  out  the  contribution  of  test  group 
expertise  to  observed  performance. 

With  regard  to  Hypothesis  2,  the  findings  provide  partial  evidence  supporting  the 
prediction  that  groups  with  a  higher  percentage  of  officers  would  deliver  a  higher  percentage  of 
food  supplies  compared  to  groups  with  fewer  officers.  Differences  in  group  skills  (group  rank, 
recent  planning  experience,  cohesion,  prior  deployments,  and  computer  experience)  were  not  a 
factor  contributing  to  collaborative  planning  success  for  face-to-face  co-located  groups. 

However,  for  groups  performing  in  the  distributed  condition,  three  group  skill  factors  (group 
rank,  recent  planning  experience,  prior  deployments)  were  related  to  higher  levels  of 
achievement,  suggesting  that  the  move  to  distributed  collaboration  might  present  additional  skill 
demands  beyond  those  required  in  co-located  planning.  These  findings  provide  evidence  that  the 
measures  of  group  expertise  used  in  the  REPSS  research  are  related  to  essential  collaborative 
planning  skills,  and  might  be  used  to  factor  out  the  contribution  of  test  group  expertise  to 
observed  performance. 

Future  ARI  research  efforts  will  likely  involve  investigating  the  potential  for  transitioning 
REPSS  to  support  the  development  of  new  communications  and  collaborative  planning 
technology  and  opportunities  to  employ  REPSS  in  a  training  development  role.  The  REPSS 
provides  a  realistic  collaborative  planning  tool  that  might  be  used  to  support  future  C2  system 
development  efforts  in  the  following  capacities: 

•  Pre-test  calibration  of  group  planning  skill  (covariate  to  assess  experiment  performance). 

Currently  the  Army  does  not  have  assessment  tools  to  estimate  the  proficiency  level  of 

groups  used  as  test  troops  for  new  systems  experimentation.  Performance  observed  in 
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new  equipment  trials  could  be  attributable  to  the  capabilities  of  the  test  troops  as  easily  as 
to  the  capabilities  of  the  equipment.  The  REPSS  can  be  used  as  a  pre-test  calibration  tool 
to  provide  an  estimate  of  group  planning  skills,  which  could  then  be  used  as  a  covariate 
in  identifying  the  relative  contributions  of  planning  skill  and  equipment  design  to 
observed  performance. 

•  Standardized  scenario  within  battle  lab  experiment.  New  systems  development  efforts 
typically  suffer  from  the  lack  of  a  standardized  scenario  that  can  be  applied  across  a 
variety  of  experiments.  Often  a  different  scenario  is  played  out  for  each  new 
experimentation  effort,  so  that  there  is  no  common  metric  for  comparison  across 
experimentation  efforts.  The  REPSS  provides  a  standardized  scenario,  with  automated 
train-up  and  performance  assessment  capabilities,  with  performance  scores  from  22 
Soldier  planning  groups  that  can  stimulate  collaborative  planning  across  a  variety  of 
technologies. 

•  Serve  as  benchmark  against  which  to  test  the  difference  (delta)  in  collective  performance 
resulting  from  a  technology  insertion. 

While  technological  innovations  are  intended  to  enhance  system  performance,  they  can 
also  present  new  demands  that  can  actually  hinder  task  accomplishment.  Development  of  future 
C2  capabilities  will  include  the  investigation  of  a  vast  array  of  technology  designs  which  need  to 
be  systematically  evaluated  to  identify  their  contribution  to  group  planning  performance.  The 
REPSS  exercise  can  serve  as  an  efficient,  baseline,  group  planning  stimulation  tool  for  use  with 
new  equipment  evaluations  allowing  designers  to  systematically  assess  the  benefits  of  selective 
technology  insertions.  Here,  the  REPSS  exercise  might  be  presented  on  a  laptop  computer  at 
each  new  equipment  node.  The  REPSS  exercise  could  serve  as  a  stimulus  for  information  that 
must  be  exchanged  across  the  new  equipment,  as  well  as  providing  an  automated  performance 
scoring  capability.  Running  a  REPSS  exercise  first  to  baseline  performance  with  a  system,  and 
then  running  the  exercise  again  with  a  new  technology  insertion,  can  provide  an  empirical 
estimate  of  the  planning  performance  difference  (delta)  associated  with  the  introduction  of  the 
new  technology. 

•  Post-test  assessment  of  group  planning  skill  (delta  on  collaborative  planning 
performance).  The  use  of  pre-  and  post-  training  REPSS  assessments  of  group  planning 
skill  could  be  employed  to  assess  the  impact  of  training  on  group  collaborative  planning 
performance. 

•  The  REPSS  should  be  able  to  be  expanded  from  Combat  Support  to  Combat  Arms  and 
Combat  Service  Support  scenarios.  Currently  the  REPSS  scenario  presents  a  simplified 
humanitarian  relief  Combat  Service  Support  operation.  This  scenario  actually  represents 
a  very  thin  “skin”  of  text  description  layered  over  a  group  collaborative  planning 
framework.  By  changing  the  text  description  of  the  scenario,  and  the  text  of  the  message 
traffic,  the  REPSS  exercise  could  be  modified  to  present  a  broad  range  of  planning 
situations.  The  planning  required  might  be  equally  representative  of  the  challenges 
facing  Combat  Arms  planners  in  a  platoon-level  mounted  combat  patrol,  or  Combat 
Support  planners  providing  Military  Intelligence  support. 
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The  REPSS  exercise  might  also  be  employed  as  a  half-day,  inexpensive  training  event 
that  could  be  conducted  as  frequently  as  required  to  maintain  basic  skills  in  collaborative 
planning.  If  desired,  the  exercise  could  also  be  used  as  a  means  to  train  unit-specific  tactics, 
techniques,  and  procedures  for  collaborative  planning.  As  previously  noted,  the  REPSS  exercise 
was  designed  to  incorporate  METT-TC  factors  into  the  tasks  participants  perform.  The  REPSS 
exercise  addresses  key  skills  that  should  be  a  prerequisite  for  success  in  any  planning  task.  Time 
management,  critical  thinking,  and  communication  skills  should  be  essential  to  all  planning 
tasks,  and  these  skills  might  be  developed  with  a  simplified  resource  allocation  task,  rather  than 
requiring  a  full  fidelity  exercise  requiring  extensive  doctrinal  and  subject  matter  knowledge. 
Automated  performance  reports  serve  as  unobtrusive  during-action  review  aids,  and  support 
post-exercise  After  Action  Reviews.  The  REPSS  simulation  serves  to  demonstrate  techniques 
that  can  be  used  to  create  a  low  cost  training  for  collaborative  planning. 

One  opportunity  for  future  REPSS  research  would  involve  assessing  the  impacts  of 
feedback  interventions  (during  and  after  the  planning  actions)  on  group  planning  performance 
success.  The  analysis  and  scoring  of  REPSS  solutions  is  performed  by  automated  measures  built 
into  the  REPSS  software,  so  that  an  overall  planning  performance  success  score  and  a  record  of 
team  planning  decisions  are  available  after  each  of  the  four  weekly  convoy  plans  are  submitted 
by  the  group.  Rather  than  simply  focusing  on  identifying  planning  coordination  mistakes,  the 
feedback  interventions  could  address  how  well  the  planning  group  integrates  METT-TC 
doctrinal  considerations  into  their  planning  solutions. 

The  REPSS  was  developed  by  the  ARI  to  be  a  tool  that  objectively  measured  group 
problem-solving  abilities,  and  can  serve  as  a  benchmark  planning  task.  The  REPSS  task 
incorporates  essential  battle  command  skills  requiring  effective  group  interpersonal 
communications,  the  ability  to  adapt  to  change,  and  a  focus  on  METT-TC  factors  influencing 
decision  making.  Results  suggest  that  the  REPSS  might  meet  the  requirement  for  a  tool  to 
support  the  development  of  the  emerging  technology,  as  it  demonstrates  sensitivity  to 
manipulations  in  planning  task  conditions,  and  planning  group  skills,  both  of  which  are  critical 
factors  impacting  planning  performance. 
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Appendix  A 
Acronyms 


ARI 

ATO 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
Army  Technology  Objective 

C2 

CITMOR 

Command  and  Control 

Citizen  Morale 

DARPA 

Defense  Advanced  Research  Projects  Agency 

FCS  C2 

FM 

Future  Combat  Systems  -  Command  and  Control 

Field  Manual 

HSU 

Humanitarian  Supply  Unit 

METT-TC 

MDMP 

Mission,  Enemy,  Terrain,  Troops,  Time,  Civilians 

Military  Decision  Making  Process 

NCO 

Non  Commissioned  Officer 

PRSD  SQRT 

Percentage  Required  Supplies  Delivered,  Square  Root 

REPSS 

Reactive  Planning  Strategies  Simulation 
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Appendix  B 
Plan  Synchronization 


Plan  Synchronization  Measurement 

The  REPSS  weekly  spending  data  provide  estimates  of  how  well  the  group  is  able  to 
synchronize  the  spending  plans  for  the  supply,  transportation,  and  security  teams.  The  group 
plan  is  considered  to  be  synchronized  when  all  HSU  purchased  and  security  guards  hired  ean  be 
transported  with  the  trucks  rented,  with  no  deficiency  or  excess  carrying  capacity.  A  perfect 
match  would  yield  a  score  of  1 00%,  while  any  percentage  of  trucks  under  or  over  the  required 
number  represents  a  waste  of  assets,  and  a  failure  of  the  teams  to  synchronize  their  plans  (10% 
too  few,  and  1 0%  too  many  would  both  yield  a  synchronization  score  of  90%).  The  scores 
reflect  the  ability  of  a  planning  group  to  create  a  synchronized  plan  in  40  minutes  for  Week  1. 

The  synchronization  score  also  reflects  the  ability  of  a  group  to  adjust  their  plan  in  20  minutes 
for  Weeks  2-4  in  response  to  changing  environmental  conditions,  and  decreasing  funds.  One 
advantage  of  the  synchronization  estimate  is  that  it  provides  an  estimate  of  collaborative 
planning  proficiency  even  if  very  few  funds  are  available.  Caution  must  be  exercised  in 
comparing  weekly  mean  synchronization  scores,  particularly  for  later  weeks,  as  only  a  few 
highly  successful  groups  might  be  contributing  to  the  mean.  It  should  be  noted  that  plan 
synchronization  is  a  necessary,  but  not  sufficient,  condition  for  performance  success.  Groups 
will  not  be  successful  if  their  individual  supply,  transportation,  and  security  team  plans  are  not 
synchronized.  However,  the  groups  can  produce  bad  plans  that  are  perfectly  synchronized, 
which  will  also  result  in  poor  performance.  The  REPSS  software  automatically  calculates  a  plan 
synchronization  score  for  each  of  the  four  towns  and  each  of  the  four  weeks.  The  scores  can  be 
provided  to  participants  as  feedback  immediately  after  each  weekly  planning  session.  The 
synchronization  scores  can  be  used  to  quickly  identify  where  problems  occur  in  group  planning, 
and  support  the  detailed  analysis  of  behaviors  contributing  to  observed  performance. 

The  group  planning  synchronization  score  provides  an  estimate  of  how  effectively  a 
group  can  coordinate  the  purchase  of  supply,  transportation,  and  security  resources  required  for 
each  town,  for  each  of  the  four  weeks.  A  score  of  100%  indicates  a  perfect  match  between  the 
carrying  capacity  of  trucks  rented  and  the  amount  of  food  and  security  guards  to  be  transported. 
The  Plan  Synchronization  scores  ranged  in  value  from  .39  to  .95,  m  =  .76,  sd=  .13.  The  plan 
synchronization  scores  will  only  have  value  in  assessing  group  performance  if  they  are  a 
contributing  factor  to  performance  outcomes.  A  Pearson  correlation  was  calculated  to  estimate 
the  relationship  between  group  plan  synchronization  scores  and  the  Percentage  of  Required 
Supplies  Delivered  -  Square  Root  (PRSD  SQRT)  scores.  A  strong  positive  correlation  was 
found  (r(20)  =  .65,  p  <  .01),  indicating  a  significant  linear  relationship  between  the  two 
variables.  This  finding  provides  evidence  that  groups  with  more  highly  synchronized  plans 
deliver  higher  rates  of  HSU,  which  supports  the  use  of  synchronization  scores  in  investigating 
group  planning  performance. 

The  examination  of  plan  synchronization  scores  provides  a  useful  approach  to  identifying 
group  planning  behavior  associated  with  performance  problems.  Detailed  plan  synchronization 
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information  was  collected  beginning  with  Group  5,  providing  information  for  a  total  of  1 8 
groups.  There  are  1 6  synchronization  scores  potentially  available  for  each  group  (four  towns  X 
four  weeks)  though  several  groups  ran  out  of  funds  and  do  not  have  Week  4  plan  synchronization 
scores.  A  review  was  conducted  to  identify  group  planning  actions  that  resulted  in  low 
synchronization  scores.  To  limit  the  number  of  cases  reviewed  it  was  decided  to  examine  scores 
that  fell  two  standard  deviations  below  the  mean  of  all  synchronization  scores.  With  m  =  .76, 
and  sd  =  .13,  the  criteria  for  review  was  set  at  a  synchronization  score  of  .50  or  lower.  Table  1 
below  describes  low  synchronization  score  cases,  by  group  planning  condition.  Where  possible, 
the  current  (low)  synchronization  score  is  compared  with  the  previous  week’s  score  (Synch 
Previous  Week/Current).  This  comparison  is  very  helpful  in  identifying  situations  where  a  group 
performed  well  in  one  week,  but  was  unable  to  adapt  their  plan  in  response  to  change  in  a 
subsequent  week. 

Plan  Synchronization  Results 

For  the  groups  performing  in  the  co-located  condition,  eight  problems  were  identified. 
With  four  of  the  seven  problems,  groups  did  well  in  Week  1  planning,  but  failed  to  coordinate  a 
major  change  in  their  spending  plan  for  Week  2  between  the  three  teams.  In  three  instances  a 
poorly  coordinated  plan  was  followed  by  a  second  poorly  coordinated  plan,  suggesting  an 
inability  to  create  a  successful  plan,  rather  than  an  inability  to  adapt  an  existing  plan  in  response 
to  changing  events.  One  low  synchronization  score  occurred  where  a  convoy  consisted  of  only 
one  truck  carrying  two  HSU.  This  left  the  single  convoy  truck  half  empty,  but  did  not  actually 
represent  a  synchronization  problem. 

For  the  groups  performing  in  the  distributed  condition,  nine  problems  were  identified. 
Two  problems  (Group  1 1  and  Group  20)  reflect  a  knowledge  failure,  where  the  participant  fails 
to  understand  that  one  truck  can  carry  four  HSU.  Three  procedure  problems  are  evident,  where  a 
group  either  failed  to  submit  their  plan,  or  submitted  only  a  partial  plan.  This  type  of  problem 
did  not  occur  for  the  co-located  groups  where  the  Commander  could  turn  in  his  chair  to  see 
whether  each  team’s  plan  included  all  four  towns,  and  that  the  plan  was  submitted  on  time.  With 
four  of  the  ten  problems,  the  groups  did  well  in  Week  1  planning,  but  failed  to  coordinate  a 
major  change  in  their  spending  plan  for  Week  2  between  the  three  teams. 

Discussion 

The  Plan  Synchronization  measure  demonstrated  sensitivity  to  planning  condition 
manipulations,  and  a  significant  shared  linear  relationship  with  the  HSU  delivery  scores.  The 
Plan  Synchronization  scores  were  used  to  focus  on  a  select  sample  of  problem  situations, 
identifying  the  types  of  performance  problems  that  led  to  poor  delivery  of  relief  supplies. 
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Table  1 


Plan  Synchronization  Problem  Review 


CO-LOCATED  PLANNING  CONDITION 


Group 

Week/ 

Synch  Score 

Problem  Description 

Town 

(previous/ 
current  week) 

7 

2  Alpha 

.86/.30 

Good  previous  week,  then  trucks  cut  by  half  but  supply  and 
security  teams  stayed  with  previous  week’s  planning  numbers. 

7 

2  Echo 

.94/.40 

Good  previous  week,  then  cut  trucks  by  63%  without  changing 

HSU  or  security  guards. 

7 

2  Oscar 

.92/.43 

Good  previous  week,  then  increased  trucks  by  56%  without 
changing  HSU  or  security  guards. 

7 

3  Alpha 

.31/.19 

Poor  previous  week,  already  had  too  many  trucks,  then  dropped 

HSU  without  reducing  trucks  proportionally. 

7 

3  Echo 

.4 1/.26 

Poor  previous  week,  too  many  trucks,  then  cut  HSU  without 
reducing  trucks  proportionally. 

7 

3  Oscar 

.43/.31 

Poor  previous  week,  too  many  trucks,  then  cut  HSU  40%  without 
reducing  trucks  proportionally. 

13 

2  Oscar 

1.0/.41 

Great  previous  week,  then  doubled  HSU  without  increasing  trucks 
proportionally.  Cut  security  assets  by  over  half. 

16 

3  Alpha 

.60/.50 

Low  synchronization  scores  both  Week  2  and  3  reflect  very  low 

HSU  quantities.  Week  3  convoy  has  only  two  HSUs  and  one  truck. 

DISTRIBUTED  PLANNING  CONDITION 

Group 

Week 1 

Synch  Score 

Problem  Description 

Town 

(previous/ 
current  week) 

5 

2  All 

.85/0 

Good  previous  week,  then  procedure  problem  where  HSU  and 

towns 

guards  requested,  but  no  request  submitted  by  transportation  team. 

9 

2  Alpha 

.83/36 

Good  previous  week,  then  added  1 0  trucks  that  were  not  needed 
while  supply  team  reduced  its  request  by  one  HSU. 

9 

2  Oscar 

.99/38 

Great  previous  week,  then  supply  team  cut  HSU  by  half,  and 
transportation  cut  trucks  75%.  Result  was  too  few  trucks. 

9 

4  Alpha 

.80/.36 

Good  previous  week,  then  doubled  HSU  and  security  guards 
request  without  proportional  increase  in  trucks. 

10 

1  EIO 

NA/0 

Week  1  procedure  problem.  HSU  and  guards  acquired  for  all 
towns,  but  trucks  only  acquired  for  town  Alpha. 

10 

3  Echo 

1.00/.34 

Great  previous  week,  then  supply  team  cut  HSU  request  by  half, 
but  transportation  team  doubled  their  request  for  trucks. 

11 

1  Alpha 

NA/.18 

Week  1  knowledge  problem.  Transportation  needed  six  trucks  but 
only  rented  one.  May  not  understand  one  truck  carries  four  HSU. 

11 

1  EIO 

NA/0 

Week  1  procedure  problem.  Transportation  and  security  provided 
assets  for  Alpha  only,  supply  team  purchased  HSU  for  all  towns. 

20 

1  and  2 

.40/.42 

Both  weeks  transportation  team  provided  one  truck  per  HSU,  not 

all  towns 

one  truck  per  four  HSU  as  required. 

Note:  The  term  NA  indicates  that  for  Week  1  problems  there  is  no  previous  week 
synchronization  score  for  comparison.  EIO  =  Towns  Echo,  India,  and  Oscar. 
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Appendix  C 
Adaptive  Planning 

Estimates  of  Adaptive  Planning  Performance 

Adaptive  planning  problem  manipulations.  The  Army  training  objective  calls  for  training 
that  fosters  adaptive  leaders,  able  to  deal  with  a  rapidly  changing  environment.  In  response,  the 
REPSS  has  embedded  problem  events  into  the  exercise  that  require  the  planning  group  to 
collaborate  and  modify  their  plans.  One  advantage  in  embedding  problems  is  that  it  provides  a 
limited  number  of  specific  events  which  can  be  scored  to  identify  where  a  group  successfully 
adapts  to  a  specific  event  requirement.  The  four  planning  problems  events  (one  for  each  week) 
require  each  group  to  gather,  exchange,  and  integrate  important  information  into  each  weekly 
plan.  Separate  pieces  of  information  identifying  a  problem  situation  were  provided  in  text 
messages,  town  newspapers,  and  with  information  that  could  be  accessed  from  the  team  maps.  A 
group  that  shared  the  information,  recognized  the  problem,  and  responded  would  allocate  their 
resources  correctly  in  response  to  the  change.  Making  the  correct  decision  for  each  problem 
event  results  in  cost  savings  and  damage  avoidance.  The  REPSS  simulation  records  the  group 
purchase  request  form  information  necessary  to  identify  whether  a  group  has  successfully 
responded  to  the  problem  situation  (ex.  convoy  route  selected  for  a  town,  type  of  supply  trucks 
rented,  the  types  of  guards  hired,  and  the  quantity  of  supplies  allocated  across  the  towns  each 
week).  The  four  embedded  problem  events  were  introduced  into  the  REPSS  task  beginning  with 
group  9,  so  that  data  are  available  for  a  total  of  14  groups  (seven  co-located,  and  seven 
distributed).  The  weekly  problem  events  are  as  follows: 

•  Week  1 :  Icy  road:  Send  unarmored  supply  trucks  on  India  Route  1  because  they  are 
more  likely  to  get  through  icy  road  conditions  than  armored  supply  trucks. 

•  Week  2:  Vulnerable  bridge:  Send  Town  Oscar  convoy  on  Route  2  with  higher  threat 
level  because  Route  1  has  a  bridge  which  would  likely  be  attacked. 

•  Week  3:  Population  shift:  Change  the  quantity  of  supplies  sent  to  each  town  in  response 
to  temporary  population  shifts  caused  by  a  regional  election. 

•  Week  4:  Civilian  guard  preference:  Hire  civilian  guards  for  Town  Oscar  instead  of 
professional  guards  because  they  will  be  better  received  by  the  population  of  this  town. 

Adaptive  Planning  Results 

Adaptive  Planning  Embedded  Problems.  The  adaptive  planning  manipulation  embedded 
one  specific  problem  situation  in  each  of  the  four  weeks  that  planning  groups  need  to  recognize 
and  respond  to  for  successful  Humanitarian  Supply  Unit  (HSU)  delivery.  The  embedded 
problems  require  that  group  members  gather,  exchange,  and  integrate  information  to  form  their 
weekly  plans.  It  was  anticipated  that  groups  in  the  co-located  condition  might  be  better  able  to 
adapt  their  plans  in  response  to  changes  in  the  environment,  as  represented  by  the  embedded 
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problems.  An  estimate  of  group  adaptive  planning  was  calculated  as  the  total  number  of 
embedded  problems  successfully  resolved.  Values  on  the  Adaptive  Planning  estimate  ranged 
from  zero  (no  problem  situations  successfully  resolved)  to  three  (three  of  the  four  problem 
situations  resolved),  (m  =  1.71,  sd  =  .99).  An  independent-samples  t  test  comparing  the 
Adaptive  Planning  scores  for  groups  performing  in  the  co-located  and  distributed  planning 
conditions  did  not  find  a  significant  difference  between  the  means  of  the  two  groups  (/( 12)  = 
1.08,  p  >  .05).  The  mean  for  the  distributed  condition  was  not  significantly  lower  ( m  =  1.43, 
sd  =  .98)  than  the  mean  of  the  co-located  condition  ( m  =  2.00,  sd  =  1 .00). 

A  Pearson  correlation  was  calculated  to  estimate  the  relationship  between  the  Percentage 
of  Required  Supplies  Delivered  -  Square  Root  (PRSD  SQRT)  measure  and  Adaptive  Planning 
scores  for  both  the  co-located  and  distributed  group  conditions.  Only  a  weak  negative 
correlation  was  found  for  the  co-located  condition  (r( 5)  =  -.05,  p  >  .10).  In  contrast,  a  moderate 
significant  positive  correlation  was  found  for  the  distributed  condition  (r(5)  =  .70,  p  <  .05). 
These  findings  provide  some  evidence  that  for  the  distributed  planning  condition,  differences 
exist  in  the  ability  of  groups  to  adapt  their  plans  in  response  to  changes  in  the  environment, 
which  contributed  to  observed  differences  in  the  delivery  of  humanitarian  relief  supplies. 

Discussion 

The  Army  Training  Plan  has  singled  out  adaptability  as  a  critical  skill  to  be  trained.  The 
REPSS  performance  measurement  approach  included  the  use  of  embedded  problems  to  detect 
each  group’s  ability  to  adapt  their  planning  solutions  in  response  to  changing  demands.  The 
embedded  problem  measurement  approach  did  identify  a  significant  relationship  between 
adaptive  planning  and  success  in  delivering  relief  supplies  for  the  distributed  planning  condition. 
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